The null hypothesis significance test in health sciences research (1995-2006): statistical analysis and interpretation

<p>Abstract</p> <p>Background</p> <p>The null hypothesis significance test (NHST) is the most frequently used statistical method, although its inferential validity has been widely criticized since its introduction. In 1988, the <it>International Committee of Medical Journal Editors </it>(ICMJE) warned against sole reliance on NHST to substantiate study conclusions and suggested supplementary use of confidence intervals (CI). Our objective was to evaluate the extent and quality in the use of NHST and CI, both in English and Spanish language biomedical publications between 1995 and 2006, taking into account the <it>International Committee of Medical Journal Editors </it>recommendations, with particular focus on the accuracy of the interpretation of statistical significance and the validity of conclusions.</p> <p>Methods</p> <p>Original articles published in three English and three Spanish biomedical journals in three fields (General Medicine, Clinical Specialties and Epidemiology - Public Health) were considered for this study. Papers published in 1995-1996, 2000-2001, and 2005-2006 were selected through a systematic sampling method. After excluding the purely descriptive and theoretical articles, analytic studies were evaluated for their use of NHST with P-values and/or CI for interpretation of statistical "significance" and "relevance" in study conclusions.</p> <p>Results</p> <p>Among 1,043 original papers, 874 were selected for detailed review. The exclusive use of P-values was less frequent in English language publications as well as in Public Health journals; overall such use decreased from 41% in 1995-1996 to 21% in 2005-2006. While the use of CI increased over time, the "significance fallacy" (to equate statistical and substantive significance) appeared very often, mainly in journals devoted to clinical specialties (81%). In papers originally written in English and Spanish, 15% and 10%, respectively, mentioned statistical significance in their conclusions.</p> <p>Conclusions</p> <p>Overall, results of our review show some improvements in statistical management of statistical results, but further efforts by scholars and journal editors are clearly required to move the communication toward ICMJE advices, especially in the clinical setting, which seems to be imperative among publications in Spanish.</p

Do multiple outcome measures require p-value adjustment?

BACKGROUND: Readers may question the interpretation of findings in clinical trials when multiple outcome measures are used without adjustment of the p-value. This question arises because of the increased risk of Type I errors (findings of false "significance") when multiple simultaneous hypotheses are tested at set p-values. The primary aim of this study was to estimate the need to make appropriate p-value adjustments in clinical trials to compensate for a possible increased risk in committing Type I errors when multiple outcome measures are used. DISCUSSION: The classicists believe that the chance of finding at least one test statistically significant due to chance and incorrectly declaring a difference increases as the number of comparisons increases. The rationalists have the following objections to that theory: 1) P-value adjustments are calculated based on how many tests are to be considered, and that number has been defined arbitrarily and variably; 2) P-value adjustments reduce the chance of making type I errors, but they increase the chance of making type II errors or needing to increase the sample size. SUMMARY: Readers should balance a study's statistical significance with the magnitude of effect, the quality of the study and with findings from other studies. Researchers facing multiple outcome measures might want to either select a primary outcome measure or use a global assessment measure, rather than adjusting the p-value

Adaptive Sampling of Information in Perceptual Decision-Making

In many perceptual and cognitive decision-making problems, humans sample multiple noisy information sources serially, and integrate the sampled information to make an overall decision. We derive the optimal decision procedure for two-alternative choice tasks in which the different options are sampled one at a time, sources vary in the quality of the information they provide, and the available time is fixed. To maximize accuracy, the optimal observer allocates time to sampling different information sources in proportion to their noise levels. We tested human observers in a corresponding perceptual decision-making task. Observers compared the direction of two random dot motion patterns that were triggered only when fixated. Observers allocated more time to the noisier pattern, in a manner that correlated with their sensory uncertainty about the direction of the patterns. There were several differences between the optimal observer predictions and human behaviour. These differences point to a number of other factors, beyond the quality of the currently available sources of information, that influences the sampling strategy

Significance testing as perverse probabilistic reasoning

Truth claims in the medical literature rely heavily on statistical significance testing. Unfortunately, most physicians misunderstand the underlying probabilistic logic of significance tests and consequently often misinterpret their results. This near-universal misunderstanding is highlighted by means of a simple quiz which we administered to 246 physicians at two major academic hospitals, on which the proportion of incorrect responses exceeded 90%. A solid understanding of the fundamental concepts of probability theory is becoming essential to the rational interpretation of medical information. This essay provides a technically sound review of these concepts that is accessible to a medical audience. We also briefly review the debate in the cognitive sciences regarding physicians' aptitude for probabilistic inference

Successful Cognitive Aging in Rats: A Role for mGluR5 Glutamate Receptors, Homer 1 Proteins and Downstream Signaling Pathways

Normal aging is associated with impairments in cognition, especially learning and memory. However, major individual differences are known to exist. Using the classical Morris Water Maze (MWM) task, we discriminated a population of 24-months old Long Evans aged rats in two groups - memory-impaired (AI) and memory-unimpaired (AU) in comparison with 6-months old adult animals. AI rats presented deficits in learning, reverse memory and retention. At the molecular level, an increase in metabotropic glutamate receptors 5 (mGluR5) was observed in post-synaptic densities (PSD) in the hippocampus of AU rats after training. Scaffolding Homer 1b/c proteins binding to group 1 mGluR facilitate coupling with its signaling effectors while Homer 1a reduces it. Both Homer 1a and 1b/c levels were up-regulated in the hippocampus PSD of AU animals following MWM task. Using immunohistochemistry we further demonstrated that mGluR5 as well as Homer 1b/c stainings were enhanced in the CA1 hippocampus sub-field of AU animals. In fact mGluR5 and Homer 1 isoforms were more abundant and co-localized in the hippocampal dendrites in AU rats. However, the ratio of Homer 1a/Homer 1b/c bound to mGluR5 in the PSD was four times lower for AU animals compared to AI rats. Consequently, AU animals presented higher PKCγ, ERK, p70S6K, mTOR and CREB activation. Finally the expression of immediate early gene Arc/Arg3.1 was shown to be higher in AU rats in accordance with its role in spatial memory consolidation. On the basis of these results, a model of successful cognitive aging with a critical role for mGluR5, Homer 1 proteins and downstream signalling pathways is proposed here

A Reckless Guide to P-values : Local Evidence, Global Errors.

This chapter demystifies P-values, hypothesis tests and significance tests and introduces the concepts of local evidence and global error rates. The local evidence is embodied in this data and concerns the hypotheses of interest for this experiment, whereas the global error rate is a property of the statistical analysis and sampling procedure. It is shown using simple examples that local evidence and global error rates can be, and should be, considered together when making inferences. Power analysis for experimental design for hypothesis testing is explained, along with the more locally focussed expected P-values. Issues relating to multiple testing, HARKing and P-hacking are explained, and it is shown that, in many situations, their effects on local evidence and global error rates are in conflict, a conflict that can always be overcome by a fresh dataset from replication of key experiments. Statistics is complicated, and so is science. There is no singular right way to do either, and universally acceptable compromises may not exist. Statistics offers a wide array of tools for assisting with scientific inference by calibrating uncertainty, but statistical inference is not a substitute for scientific inference. P-values are useful indices of evidence and deserve their place in the statistical toolbox of basic pharmacologists